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ABSTRACT 

A variety of techniques was used to assess the 
validity of the Descriptive Tests of Mathematics Skills (DIMS) for 
making placement decisions. The DTMS is a group of four tests 
(Arthmetic Skills, Elementary Algebra Skills r Intermediate Algebra 
Skills, and Functions S Graphs) that was designed to help colleges 
place each admitted student in the appropriate mathematics course. 
The DTMS were administered in freshman level mathematics courses at 
36 institutions that represented a broad spectrum of two-year and 
four-year colleges. A pre-post design was used in seven institutions, 
with a pretest only in the other colleges. Results indicated that the 
DTMS were: (1) sensitive to gains from instruction in a single 
semester; (2) highly correlated with course grades obtained 
concurrently; (3) predictive of end of course grades when the DTMS 
were administered at the beginning cf the course; (4) a better 
predictor of course grades than SAT-Mathematical scores for remedial 
courses; (5) related to course difficulty as perceived by students; 
(6) capable of demonstrating a trait- treatment interaction in a 
precalculus-calculus sequence: and (7) related to the content of 
mathematics courses a*s judged by mathematics faculty members. 
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Abstract 



A variety of techniques were used to assess the validity of the 
Descriptive Tests of Mathematics SkillB (DTMS) for making placement 
decisions. DTMS tests were administered in 36 institutions that 
represented a broad spectrum of two-year and four-year colleges. A 
pre-post design was used in seven institutions, with a pretest only in 
the other colleges. 

Results indicated that the DTMS tests were: 

(1) sensitive to gains from instruction in a single semester; 

(2) highly correlated with course grades obtained concurrently; 

(3) predictive of end of course grades when the DTMS tests were 
administered at the beginning of the course; 

(A) a better predictor of course grades than SAT-Mathematical scores 
for remedial courses; 

(5) related to course difficulty as perceived by students; 

(6) capable of demonstrating a trait-treatment interaction in a 
precalculus-calculus sequence; and 

(7) related to the content of mathematics courses as judged by 
mathematics faculty members. 
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I . Introduction 

The primary purpose of the Descriptive Tebta of Mathematics Skills 
(DTMS) ia M to aaaiat colleges in the proper placement of admitted 
atudenta within the sequence of mathematics courses offered by a given 
institution 11 (Jones, 1977). There are four DTMS tests: a 35-item 
Arithmetic Skills test, a 35-item Elementary Algebra Skills test, a 
30-item Intermediate Algebra Skills test, and a 30-item Functions and 
Graphs Test. All tests are in a four-choice multiple choice format, and 
the testing time is 30 minutes for each test. Each test contains three 
or four descriptive clusters that represent groupings of similar items 
(e.g., operations with fractions, coordinate plane, and graphs). Complete 
test descriptions and technical data are available in the Guide to the 
Use of the Descriptive Tests of Mathematics Skills (College Board, 1979). 

Cronbach (1971) has noted that it is not really a test that is 
validated but rather "an interpretation of data arising from a specified 
procedure 11 (p. 447). Thus, each decision made from each of the DTMS 
tests must be validated separately. For example, the Arithmetic Skills 
test might be valid for placing students into a course emphasizing 
arithmetic computation but not a course in applied arithmetic. Similarly, 
it might be valid for the arithmetic computation course at one school, 
but not at another school either because the courses differ, the students 
differ, or a combination of the two. With the number of courses open to 
beginning students (Jones & Williams [19 74] identify 17 courses through 
the calculus) and the variety of two-year and four-year colleges, any 
truly comprehensive validation study is practically impossible. Each 
DTMS user should conduct its own validation study. The purpose of this 
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stucy, then, was to determine whether the DTMS tefltH possess character 1st lea 
that are generally useful for tests used to make placement decisions. 
Areas investigated included analyses of content validity, sensitivity to 
instructional gains, concurrent validity, predictive validity, relationship 
to student perceptions of course difficulty level, and trait-treatment 
interactions. Each of these areas is a separate section in this report, 
and the rationale behind each of the analyses is presented in more detail 
in the appropriate section. 
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2. Matluul 

Samplu Solection 

CrlterLa for auleet Low of hint Uuttona wore «h follows* 

(1) Broad geographical distribution (at Umst ono institution 
from each group of states served by a Collage Hoard 
regional office). 

(2) Both two-year and four-year institutions. 

(3) Range of selectivity from fairly selective to open admissions. 
Initial contacts were made by College Board field staff to institutions 

they thought might be willing to cooperate. Thus, the sample recruited 
represented institutions that were quite diverse on a number of dimensions, 
but it was not a random sample. Brief descriptions of the 36 participating 
institutions are presented in Appendix A. Colleges are identified by 
code letters; double letter codes (e.g., College AA) are used to identify 
two-year colleges, and single letter codes identify four-year colleges or 
universities. Basic data is from the 1977 College Handbook (CEEB, 1977). 
Tne selection ratio is the ratio of students accepted to students applying. 

Each institution was asked to administer the tests either to all 
freshmen or to all students enrolled in mathematics courses that were 
open to freshmen from the most elementary course through calculus. The 
latter option was consistently chosen because it was logistically much 
simpler. 



Pyuicgduraa 

For thin pro.jaet or. ly, the tour taata wave printed In a a log I a 
book accompanied by a a Ingle common anawar aheet . Inatruetlona on the 
teat booklets aaked atudenta to take the two of tha four teata that wera 
moat relevant to their high school mathematlca preparation. Students 
with no more than one year of high achool algebra were naked to take 
Arithmetic Skills and Elementary Algebra Skilla teats; atudenta with more 
than one year's study in algebra but not trigonometry were aaked to take 
the Elementary Algebra Skills and Intermediate Algebra Skilla testa; 
students with more than one year of algebra and at least one-half a 
semester of trigonometry were asked to take the Intermediate Algebra Skills 
and Functions & Graphs tests. The front of the test answer sheet 
requested some basic background information from each student including 
sex, birthdate, number of semesters in various high school mathematics 
courses, and the grades achieved in these courses. 

Testing was at the beginning of the fall semester in 1978. At the 
end of the fall semester, the participating colleges were asked to supply 
final grades in the mathematics courses for all students who took the 
fall tests. In addition, posttest data were obtained from seven colleges 
that readministered the DTMS tests at the end of the semester. This more 
intensive data collection in a limited number of schools was planned and 
did not reflect a high drop-out rate among the other institutions. 

Additional data collection procedures that were unique to particular 
analyses (e.g., the course satisfaction questionnaires and the content 
analysis) are described in the relevant chapters. 




ir a tuct u Me\\vAt*\y tardea to th* turnout of a r.nii^ Uiut 
assuming ln«tnirt ton In that muv** U at Ua^t. *om*wlicU ettmUva), 
then the teat whoiiUI be AtuinUlvt* to ^alua lu knowledge fvom the beginning 
of the course to the end. Gain analysis u particularly relevant for 
testa which may le used to exempt students from course requirements 
(Will Ingham, 19 74) • 

Although analysis of score gain U a potent tally valuable method 
of empirically hhnuhh log tuHt validity, mich analyses must bu Interpreted 
cautiously na there arts a number of Morions problems In assessing raw 
gainH (Harris, 1963). Regression to the mean, for example, could produce 
spurious gains tn a group that was placed into a remedial section on the 
basis of low tost scores. However, since DIMS scores were not used 
for selection in the current study, the portion of the regression due 
to errors of measurement on that instrument was eliminated. Furthermore, 
given a highly reliable selection test and reasonably high correlations of 
pretest and posttests, regression effects should not be too large, and 
should be essentially nonexistent for middle-ability students who are 
self -selected into typical courses. 

Inflated gain estimates also could result from students who find the 
answers to specific items from the pretest and remember these answers on 
the posttest. If, as a result of the pretest experience, the student 
finds out how to solve problems of the same type that caused difficulty, 
this should be considered as real learning; it would be a spurious gain 
only when learning a specific correct answer did not generalize to 
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similar items in the same content domain. In the current study, this 
type of gain was reduced by not allowing the students to see the questions 
or their answers after the pretest administration. In addition, students 
knew that they would not be graded on their gains on the DTMS and, hence, 
were not motivated to memorize answers to specific questions. Neverthe- 
less, some practice effect was undoubtedly present, and small gains must, 
therefore, be interpreted skeptically. 

Gain analyses will be presented for courses in arithmetic, two 
different levels of algebra (elementary and "college" algebra), and 
precalculus. 

Arithmetic Courses . Pretest and posttest DTMS scores were available 
for arithmetic courses from two institutions. The course in a two-year 
community college in a large southwestern city (College LL) was described 
as a course in prealgebra mathematics "designed to develop an understanding 
of fundamental operations using whole numbers, fractions, decimals, and 
percentages and to strengthen basic skills in mathematics . . • [it] 
includes an introduction to algebra." Students in this course generally 
had less than one year of algebra in high school. Of the 44 students 
included in the analysis for College LL, 35 reported a grade in high 
school general mathematics and 29 reported an algebra grade. On a 4.0 
point scale (i.e., A-4.0, B=3.0, etc.) the average general mathematics 
grade was 2.1 and the average algebra grade was 1.8. Slightly over 90% 
of the students were white, and the sexes were equally represented. 

The other course with a major arithmetic component was designed for 
special admissions students at a large midwestern university (College L) . 
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Half of this course was a review of basic arithmetic skills while the 
second half introduced elementary algebra concepts. Students in this 
course generally had one year of high school algebra. For the 77 (out of 
109) students who reported high school algebra grades, the mean was 2*2 
on a 4.0 point scale. About 57% of the students in this group were female 
and about 60% were white. 

Pretest and post test means and standard deviations and gains for the 
Arithmetic Skills and Elementary Algebra Skills tests are presented in 
Table 3.1. 

Table 3.1 

Gains on the Arithmetic Skills and Elementary Algebra 
Skills Tests for Remedial Arithmetic Courses 



Test 



College LL 
M SD 



College L 
N M SD 



Arithmetic Skills - 
pre 

Arithmetic Skills - 
post 



44 



22.45 5.56 



27.84 4.71 



109 



25.44 5.41 



27.50 4.93 



Gain 



5.39 



2.06 



Elementary Algebra - 
pre 

Elementary Algebra - 
post 



40 



10.28 4.98 



10.80 6.52 



108 



12.88 5.30 



16.68 6.78 



Gain 



.52 



3.80 
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Students in both colleges apparently gained in the kinds of skills 
assessed by the Arithmetic Skills test. Although the end of course 
arithmetic skill level was nearly identical in the two colleges, students 
in College LL showed more gain because they started at a lower level* 
Comparison of these DTMS gains with gains on the Test of Standard Written 
English (TSWE) (Breland, 1977) suggests that the arithmetic skills 
assessed with DTMS may be more susceptible to change in a single course 
than are the writing skills assessed by TSWE* Of four schools in Breland's 
report, the largest gain was about .35 of a wi thin-group standard 
deviation and the smallest gain was about • 15 of a within-group standard 
deviation as compared to the DTMS gain of from .4 to a full standard 
deviation. 

Analysis of the gain in each cluster of the Arithmetic Skills test 
presented in Table 3.2, reveals an interesting pattern. 



Table 3.2 

Cains on Each or Four Clusters on the Arithmetic Skills Test 



College LL (N - 44)_ 



Pretest Mean (SD) 
Pojttest Mean (SD) 



Cluster A 
8.05 (1.01) 
8.59 ( .62) 



Cl uster B 
5.09 (2.82) 
7.93 (1.97) 



Cluster C 
5.41 (2.12) 
7.11 (2.09) 



Cluster D 
3.91 (1.70) 
A. 20 (1.53) 



Cain 



.54 



2.84 



1.70 



.29 
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Col lege L (N - 1091_ 



Cluster A Cluster B Cluster C Cluster U 

8.28 ( .92) 7.33 (2,50) 5.62 (2.33) 4.22 (1.44) 

8,30 ( .89) 8.31 (2.22) 6.59 (2.14) 4.30 (1.37) 

.02 .98 .97 .08 
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Although gains are consistently higher in College LL (as was indicated on 
the total sum gains), the pattern of gains is remarkably consistent in 
the two institutions • Cluster A (operations with whole numbers) contains 
nine items and scores in both schools are at ceiling levels as reflected 
by the high means and reduced standard deviations relative to the other 
cluster scores. Thus, mean gains in this cluster are necessarily small. 
The largest gains are found in Cluster B (ten items on operations with 
fractions) and Cluster C (ten items on operations with decimals and 
percents). The small gains in Cluster D (six items on simple applications 
involving computation) may again reflect some test ceiling problems 
(though not as severe as in Cluster A) but may also reflect some £ *v 
difficulty in teaching these skills in a single semester course. Pretest- 
posttest correlations also reflect the lack of meaningful variation on 
Cluster A but indicate relatively stable measurement with Cluster D. 
In College LL, the pre-post correlations were .03, .31, .59, and .56 for 
Clusters A-D, respectively, while the comparable values in College L 
were .11, .48, .49, and .52. 

The gains on the Elementary Algebra Skills test indicate that the 
half semester introduction to elementary algebra concepts at College L 
was considerably more effective than the "introduction to algebra" of 
unspecified duration at College LL. However, this should not be construed 
as a criticism of College LL, where limited resources were effectively 
used in improving the more basic arithmetic skills. This small gain also 
suggests that using the same form for pre- and posttests need not necessar- 
ily result in large gains, thus increasing the credibility of the other 
gains reported. 
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Willingham (1974) notes that analysis of score gains for students 
who earned different grades in a course may be one way of demonstrating 
the extent to which "the test confirms the teacher's judgments — and vice 
versa" (p. 164). Score gains for each final course grade in College LL are 
presented in Table 3.3. Final grades were not available from College L. 



Table 3.3 

Gains on the Arithmetic Skills Test 
by Course Grades at College LL 



Grade in Arithmetic Skills Arithmetic Skills 

Course N Pretest Posttest Gain 

A 17 26.0 31.3 5.3 

B 17 21.5 26.6 5.1 

C 6 19.0 25.8 6.8 

D 4 16.5 21.3 4.8 

F 0 
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Unlike the 1973 Feldman and Kane study cited by Willingham (197A), 
gains were not clearly related to course grades. However, it is important 
to note a critical difference between the studies. Students in the 
calculus course described by Feldman and Kane had little or no direct 
instruction in calculus before the course began, and pretest scores of 
students who eventually got As or Bs were little different from those 
of students who got Ds or Es. On the other hand, students in the current 
study presumably studied arithmetic skills for many years in secondary 
schools before they took the pretest in college. Posttest scores are 
largely reflective of these preexisting differences rather than indicating 
differential score gains as the result of instruction. Correlational 
evidence paints just about the same picture; the correlation of pretest 
scores with grades is nearly as high as the correlation of posttest 
scores with grades (.59 versus .64). Grading standards at most institu- 
tions are designed to reward final status and not gain. But these data 
are a good reminder that C students may work just as hard and benefit 
just as much from instruction as A students. 

Elementary Algebra Courses . Two colleges with courses in elementary 
algebra reported data for the gain analysis. One college was the large 
midwestern university that also provided scores for the above arithmetic 
analysis (College L). Special admissions students who scored above a 
minimal level on a locally developed placement test were placed into the 
elementary algebra course rather than the developmental mathematics 
course described above. On the average, students in this course had 
about one year of algebra in high school. Thirteen of the sixteen 
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studentfc reported grades in high school algebra, and the average grade 
was 2.38 (out of 4.0). 

The other institution reporting data for the analysis was a two-year 
public community college in a large southeastern metropolitan area 
(College JJ). The sample was about 60% female and 80% white. On the 
average, students had one year of high school algebra. Grades in high 
school algebra were reported by 102 of the 132 students; the average 
grade reported was 2.02. 

Pretest and posttest means and standard deviations for the Elementary 
Algebra Skills test are presented in Table 3.4. 



Table 3.4 

Gains on the Elementary Algebra Skills Test for Elementary Algebra Students 



Elementary Algebra (pre) 
Elementary Algebra (post) 



N 



16 



College L 

M SD 



College JJ 

M SD 



9.13 
14.13 



3.91 
6.28 



132 



11.95 
19.42 



5.12 
6.38 



Gain 



5.00 



7.47 
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Substantial gains were evident in both courses. In College L, 
pretest scores were close to the level that could be achieved by random 
guessing (random guess level » number of items [35] - number of 
choices per item [4] ■ 8.75). The increased mean and variance in the 
posttest scores suggests that they provide more meaningful measurement. 
In both colleges, the posttest scores are still quite low relative to the 
maximum possible score of 35. This might be due to the fact that no 
instruction was provided for many of the skills assessed on the test 
(suggesting that much of the Elementary Algebra Skills test was not a 
valid measure for these courses) or it might be that the test provides a 
valid indication of the fact that many students failed to learn a number 
of the skills taught. The analysis of gains by course grade in Table 3.5 
helps to resolve these conflicting interpretations. 

Table 3.5 

Gains on the Elementary Algebra Skills Test 
by Course Grades at College JJ 

Grade in Elementary Algebra Elementary Algebra 



Course N Skills Pretest Skills Posttest Gain 

M _SD M _SD 

A 14 15.9 4.5 26.9 3.3 11.0 

B 31 14.8 5.9 23.8 3.4 9.0 

C 28 13.0 3.9 21.5 3.7 8.5 

R* 58 8.9 3.5 14.2 5.0 5.3 



*No Ds or Fs given in remedial courses; grade of R means that course must be 
repeated. 
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Since A students answered an average of 26.9 items correctly, it 
seems likely that most of the skills on the Elementary Algebra Skills 
test were taught at some level, but that many of these skills were 
learned by only the best students. The high correlation between scores 
on the Elementary Algebra Skills posttest and a composite score reflecting 
the average of class tests and points given for classwork (r » .78) also 
suggests that the Elementary Algebra Skills test measures algebra skills 
that are relevant to the specific course. 

Unlike the gains on the Arithmetic Skills test, gains on the Elementary 
Algebra Skills test are systematically related to course grades achieved. 
Nevertheless, pretest scores of students who ended up having to repeat the 
course were substantially below pretest scores of students who eventually 
got As. 

The analysis of gains in each cluster in Table 3.6 shows the greatest 
gains in Cluster B ( 17 items on operations with algebraic expressions). 



Table 3.6 

Gains on Each of Three Clusters on the Elementary Algebra Skills Test 

College L» College JJ** 

Cluster A Cluster B Cluster C Cluster A Cluster B Cluster C 

Pretest Mean (SD) 2.88 (1.63) 3.31 (1.66) 2.94 (1.65) 3.73 (1.75) 5.14 (2.94) 3.08 (1.73) 

Posttest Mean (SD) 4.25 (2.24) 6.69 (3.42) 3.19 (1.64) 5.02 (1.97) 9.73 (3.68) 4.67 (1.86) 



Gain 1.37 3.38 .25 1.29 4.59 1.59 

*N - 16 
**N - 132 
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Gains on Cluster A (9 items on operations with real numbers) and Cluster 
C (9 items on solution of equations, inequalities, and word problems) 
were less dramatic; indeed there was probably no real gain in Cluster C 
at College L. 

College Algebra Courses * Three institutions with college algebra 
courses provided scores for this analysis. College HH is a two-year 
public community college in the suburbs of a major midwestern city. All 
of the students in the current analysis from College HH were white and 
54% of them were male. Most of them had two years of high school algebra, 
and the average high school algebra grade was 3*20 (for the 10 out of 11 
students who reported grades). College E is a four-year public college 
in a small eastern town. The students in the current analysis were 95% 
white and 55% female. Half of the students reported that they had 
one year of high school algebra and the other half reportedly had two 
years. The average grade for students with one year of algebra was 2.8, 
while the students with two years of algebra reported average grades of 
2»5 for the second year course. College LL is a two-year public community 
college in a large southwestern city. Of the students in the college 
algebra course 73% were male and 95% were white. Most of the students 
had two years of high school algebra, and the average grade was 3*02 
(with all of the 41 students in the analysis reporting a grade). 

As indicated in Table 3«7, pretest differences among the three 
colleges on the Intermediate Algebra Skills test were fairly substantial. 
The relatively low mean and high standard deviation in College E probably 
is a result of the substantial number of students there who had only one 
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year of high school algebra. Posttest scores, on the other hand, were 



Table 3.7 

Gains on the Intermediate Algebra Skills Test for 
College Algebra Students in Three Colleges 

College HH College E College LL 

N M_ SD Nil SJD N ' SD 

Intermediate Algebra 

Skills (pre) 19.18 4.73 12.84 6.60 14.85 5.09 

11 19 41 

Intermediate Algebra 

Skills (post) 24.18 3.89 22.37 4.81 20.54 4.14 

Gain 5.00 9.53 5.69 



more comparable, indicating some communality in course content even 
though this presumably required more learning for students in College E. 
Gains in all three institutions were substantial (i.e., greater than one 
standard deviation). The magnitude of these gains can be better evaluated 
by comparing them with gains of students with similar pretest scores on 

23 
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the Intermediate Algebra Skills test who were enrolled in math courses 
that did not emphasize instruction in algebra, although some algebraic 
concepts may have been included in these other courses. Specifically, 
gains on the Intermediate Algebra Skills test for one set of students 
from College E who enrolled in a course in business mathematics and a 
second set of students who enrolled in a probability and statistics 
course were evaluated. Although very little confidence can be placed 
in these results due to the very small sample sizes, the analysis of 
gains in these courses is included in this report as an illustration 
of the kind of comparative analysis that may be useful in local 
validation studies. As indicated in Table 3.8, gains were considerably 
smaller in these courses. Gains in the probability and statistics course 



Table 3.8 

Gains on the Intermediate Algebra Skills Test for Business Math 
and Probability and Statistics Students at College E 

Probability & 
Business Math Statistic s 

N M SD N M SD 

Intermediate Algebra 

Skills (pre) 18.45 6.88 19.17 4.34 

11 6 

Intermediate Algebra 

Skills (post) 20.27 6.93 21.83 4.36 

Gain 1.82 2.66 
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were still over half a standard deviation. However, analysis of the 
cluster scores indicated that nearly all of this gain could be accounted 
for by the third cluster (10 items on the coordinate plane and graphs). 
Scores on that cluster were 4.83 at pretest and 7.00 at posttest for a 
gain of 2.17. For students in the college algebra courses, gains were 
consistent across all three clusters. 

Precalculus Courses , Pretest and posttest DTMS scores were available 
for precalculus courses from two institutions. At college E (four-year 
public college in a small eastern town), the course was titled "Precalculus 
Mathematics." Most of the students reported that they had two years of 
high school algebra. The average algebra grade was 3.3. The sample was 
56% male and 89% white. The second college reporting scores was a large 
private university in a major western city (College T) . The course was 
titled "Introductory College Mathematics" and is listed as a prerequisite 
for the Calculus I course, although students also could qualify for 
Calculus I with high school courses in trigonometry and analytic geometry. 
The course content included sets, functions (including exponential, 
logarithmic, and trigonometric functions) graphing, systems of linear 
equations, and analytic geometry. Most students in the course reportedly 
had two years of high school algebra, with an average grade of 3.4 
reported. The sample was 57% male and 56% white. 
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Pretest and posttest means and standard deviations and gains for 
both the Intermediate Algebra Skills and the Functions & Graphs tests are 
presented in Table 3.9. 



Table 3.9 

Gains on the Intermediate Algebra Skills and Functions & Graphs 
Tests for Precalculus Students in Two Colleges 

College E College T 

N M SD N M SD 

Intermediate Algebra 

Skills (pre) 18.69 4.90 19.98 4.33 

35 127 

Intermediate Algebra 

Skills (post) 24.43 3.42 24.78 3.75 



Gain 5.74 4.80 



Functions & 

Graphs (pre) 14.94 4.15 16.16 5.09 

34 81 

Functions & 

Graphs (post) 23.50 3.54 24.53 3.39 

Gain 8.56 8.37 
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On both tests, gains were remarkably similar in the two colleges. The 
gain of about two standard deviation units on the Functions & Graphs test 
was especially striking. Analysis of gains on the individual clusters of 
the Intermediate Algebra Skills test was not particularly enlightening, 
with approximately equal gains on all clusters at both institutions. 
However, an interesting pattern emerged with the Functions & Graphs 
cluster scores. As indicated on Table 3.10, gains on Cluster C (13 items 
on trigonometric functions) appeared to be especially large. An average 



Table 3.10 

Gains on Each of Three Clusters on the Functions & Graphs Test 

College E* College T** 

Cluster A Cluster B Cluster C Cluster A Cluster B Cluster C 

Pretest Mean (SD) A. 35 (1.57) A.15(1.79) 6.44(2.49) 5.49(1.79) 4.35(1.82) 6.30(2.69) 

Posttest Mean (SD) 6.20 (1.30) 6.26 (1.60) 11.03 (1.66) 6.59 (1.29) 6.77 (1.74) 11.17 (1.45) 



Gain 1.85 2.11 4.59 1.10 2.40 4.87 



*N - 34 
**N - 81 
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of less than half of these Items were answered correctly on the pretest, 
but by the posttest the average score was less than two points from the 
total possible. The reduced standard deviation on the posttest Is an 
additional Indication that the test celling had been reached* 

An analysis of gains on the Intermediate Algebra Skills test classified 
by course grade Is presented In Table 3 • 1 1 • As was noted with the similar 



Table 3.11 

Gains on the Intermediate Algebra Skills Test 
by Grades for Precalculus Students 



College £ College T 







Pretest 


Posttest 






Pretest 


Posttest 




Grade in 
Course 


N 


M 


SD 


M 




SD 


Gain 


N 


M 


SD 


M 


SD 


Gain 


A 


7 


24.0 


3.9 


27. 


0 


2.0 


3.0 


30 


22.8 


3. A 


27.8 


1.6 


5.0 


B 


13 


19.2 


A. 7 


25. 


,8 


3.2 


6.6 


26 


20.7 


A. A 


25.9 


2.8 


5.2 


C 


8 


16.1 


3. A 


22. 


,3 


2. A 


6.2 


35 


18. A 


3.6 


2A.2 


3.1 


5.8 


D 


6 


14.7 


2.3 


22. 


,2 


3.2 


7.5 


23 


18.5 


A. 7 


22.0 


3.9 


3.5 


F 


1 


20.0 




20, 


.0 






7 


17.9 


A. 9 


19.6 


A. 2 


1.7 
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analysis of gains on the Arithmetic Skills test, gains are not systemati- 
cally related to grades assigned although grades are systematically 
related to posttest scores (r = .60 at College E and .68 at College T). 
It also should be noted that the Intermediate Algebra Skills test has 
only 30 items and gains in the "A" category might well have been greater 
if the test ceiling were not encountered. Analysis of gains by grade for 
the Functions & Graphs test presents an almost identical situation, and 
therefore will not be presented here. 

Summary 

The DTMS tests appear to be sensitive to instructional gains in 
college courses in arithmetic, elementary algebra, college algebra, and 
precalculus mathematics. They are probably poorer at assessing gains in 
other skill areas for which they were not specifically designed (e.g., 
business math and statistics). Analysis of cluster scores indicates 
certain clusters of items within a test might be more sensitive to gains 
in particular courses that emphasize those skills. Although within 
school sample sizes were frequently quite small, the replicability of the 
findings across institutions permits considerable confidence in the 
results. 
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4. Concurrent Validity 



Where Course X is a prerequisite for Course Y, a placement test 
should be able to show when a student has already mastered the content 
of Course X and is, therefore, ready to go directly to Course Y* An 
indication of the validity of the placement test, then, can be determined 
by its correlation with an indicator of success in Course X* A simple 
pass-fail classification could bt - ich a success indicator, but that 
classification would not reflect differing degrees of success that might 
be of interest. Course grades provide not only pass-fail information, 
but also indicate relative success among those who pass. Thus, scores on 
the appropriate test of the DTMS administered at the end of a course were 
correlated with end of course grades. Grades were coded A ■ 4, B « 3, 
etc. Students who withdrew from a course with an indication that they 
were failing at the time were asigned Os, but students who simply withdrew 
with no indication of success or failure were deleted from these analyses. 
At College JJ, no grade lower than a C was given in remedial courses, but 
a "non-punitive" grade of R could be assigned indicating the student had 
to repeat the course. R grades were coded as 0* When available, numerical 
scores representing points accumulated over the semester or scores on a 
comprehensive final examination were also used as criterion variables as 
they permit a greater score range, hence possibly higher correlations, 
than do grades. An additional advantage of these scores is that they can 
be used to refute claims of possible criterion contamination in the 
grades. The correlation between grades and DTMS scores could be spuriously 
high if some faculty members assigned grades based partially on their 
students' performance on the DTMS posttest. But this contamination could 
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not apply to numerical scores that were assigned based on performance 
that was completely independent of the DTMS. Even with the grades, 
serious criterion contamination seems unlikely. Scores were not reported 
to the institutions before final grades were received. Although some 
faculty members could have hand-scored the answer sheets and used these 
to partially determine grades, an informal survey indicated that this was 
not done. 

Because more than one DTMS test is relevant for some courses and 
because better correlations are sometimes possible with longer, more 
reliable tests, scores from two DTMS test were added together and these 
composite scores, as well as individual subtest scores, were then corre- 
lated with the criterion scores. 

Arithmetic Courses . At College LL, the correlation of the Arithmetic 
Skills test posttest score with grades was .64 (N « 45). Adding the 
Elementary Algebra Skills test score to the Arithmetic Skills test score 
cesulted in a correlation of .59. This is a good indication that a 
longer test is not necessarily better than a shorter test if the extra 
items merely add error variance; recall that the posttest Elementary 
Algebra Skills test mean was only 10.8 in this sample. At College L, 
where the course included both arithmetic review and elementary algebra, 
final grades were not reported. However, a course score indicating the 
number of points on the 50-item course final examination was available. 
The correlation of the Arithmetic Skills test score with this course 
score was .42 (N - 109). The correlation of the Elementary Algebra 
Skills test score with this course score was .53 (N » 108) and the 
correlation of the score from the two DTMS test combined with the course 
score was .55 (N «■ 108). Considering that these are remedial courses 
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serving a restricted range of student abilities, these correlations from 
short 35-item tests are remarkably high. 

Elementary Algebra Courses . At College JJ, the correlation of the 
Elementary Algebra Skills test scores with grades was .75 (N - 139) and 
the correlation with a course score reflecting an average from tests, 
class work, and the final examination was .78* This very high correlation 
suggests that the Elementary Algebra Skills test can be an excellent 
indicator of success in elementary algebra courses. 

College Algebra Courses * At College LL, posttest Intermediate 
Algebra Skills test scores correlated .75 (N * 41) with grades. The 
correlation of Elementary Algebra Skills test scores with grades was .62 
(N » 29) and the correlation for the combined score was .70. 

Precalculus Courses . The correlation of grades with Intermediate 
Algebra Skills test scores at College E was .60 (N =* 35). The correlation 
with the Functions & Graphs test score was .61 (N ■ 35), and for the 
combined tests, the correlation was .70 (N « 35). At College T, the 
correlation of grades with Intermediate Algebra Skills test scores was 
.65 (N = 131), with Functions & Graphs test scores .60 (N - 131), and 
with the combined score was .72 (N =» 131). 

A high correlation of scores on a placement test and an indicator 
of success in a course (e.g., grades) suggests that the test is a potentially 
useful device for exempting students from the course. However, the corre- 
lation by itself does not indicate where the cutting score should be. 
To establish reasonable cutting scores, a cross-tabulation of test scores 
and grades is needed. Such a cross-tabulation for the precalculus course 
at College T is presented in Table 4.1. Course grades for various levels 
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of the combined Intermediate Algebra Skills and Functions & Graphs 
tests are provided. 



Table 4.1 

Cross-tabulation of Course Grades and End of Course Scores 
from the Combined Intermediate Algebra Skills and 
Functions & Graphs Tests for a Precalculus Course 



Course 



Grades 


Scores on 


Intermediate Algebra 


Skills 


and Functions & Graphs 




0-20 


21-30 


31-40 


41-50 


51-60 


Totals 


A 


0 


0 


0 


1 


29 


30 


B 


0 


0 


2 


11 


17 


30 


C 


0 


0 


2 


24 


12 


38 


D 


0 


0 


9 


10 


4 


23 


F 


1 


3 




3 


0 


10 


Totals 


1 


3 


16 


49 


62 


131 
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Although where to set cutting scores would have to be decided upon 
locally, a reasonable approach would seem to be to encourage students 
scoring over 50 on the combined tests to skip the course. None of the 62 
students with this score received failing grades, and only 16 received 
grades lower than a B. Students on the 41-50 range also might be allowed 
to skip the course. However, such a decision would be reasonable only if 
C students in the precalculus course ^ave a fairly good success record in 
the next course in the sequence (calculus) • Scores of 40 and below 
clearly indicate very limited success in the precalculus course, and 
students with such scores probably should be required to take the course. 
These cutting scores are discussed only for illustrative purposes. 
Individual institutions might find other cutting scores more meaningful. 
They also might find that grouping scores into smaller score intervals 
provided more useful information, especially for very large courses. 

Summary 

Correlations for all courses were substantial, especially considering 
the restricted range of abilities in most of the courses and the unreli- 
ability of grades as a criterion. Evidence from the correlations and cross- 
tabulations suggests that the DTMS tests could be very useful for placement 
in a segmented sequence of mathematics courses. 
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A demonstration of predictive validity may be irrelevant for certain 
placement decisions (Willtngham, 1974; Cronbach & Snow, 1977). Indeed, 
Gronbach (1971) states the case in the extreme, asserting that "a 'valid- 
ity coefficient' indicating that Test X predicts success within a treat- 
ment tells nothing about its usefulness for placement." The point these 
authors make is that what is really needed is a demonstration of a 
tx" Lt-treatment interaction. Thus, they would argue that the regression 
line oi math aptitude (as measured on the placement test) on achievement 
(as measured at the end of Course B) should be steeper for students who 
were placed directly into Course B than for students who took Course A 
first. In other words, it is of little use to predict that students 
W)uld fail in Course B unless you can show that they would be more likely 
U succeed had they taken Course A first. 

However, demonstration of a fairly high predictive validity of the 
placement test for students in "regular" courses would be a necessary 
(though not sufficient) condition for the later demonstration of a 
trait-treatment interaction. If the regression slope relating DTMS scores 
to math achievement were not fairly steep in the "regular" (or short 
sequence) group of students, there would be little hope of demonstrating 
a less steep slope for a remedial (or lower level) plus regular group 
(i.e., a long sequence). Furthermore, the basic assumption of a trait- 
treatment interaction study that performance in Course B should be max- 
imized for all students may not be correct in some cases. Some students 
with very low scores may elecr to enroll in programs that would never 



require them to take Course B rather than try to seek the additional 
coursework that might be necessary for them to succeed* This argument 
might not apply to English courses that would be required for all students, 
but might very well apply to calculus courses that can be completely 
avoided by students enrolled in certain programs. 

Information on predictive validity might also be useful to a 
faculty advisor or counselor even if no formal remedial course sequence 
were available. If a valid placement test predicted that a student would 
have trouble even in the lowest level course, the advisor might suggest 
arranging other courses taken that semester so that a maximum amount of 
effort could be expended on the course where the greatest difficulty were 
predicted. Although knowledge of predictive validity may not be the 
ultimate or only indication of the worth of a placement test, it should 
still provide some valuable information. 

Method 

For assessing concurrent validity, the most relevant test would be 
one that closely matched the content of the course. But for assessing 
predictive validity, relevance should be determined in relation to 
prerequisite skills for the course. For example, a calculus test 
should not be used in determining predictive validity for a beginning 
calculus course because students would not be expected to be able to 
answer any calculus questions at the beginning of the course. However, a 
test of functions and graphs would be appropriate as these are important 
prerequisite skills for the study of calculus. 

Pretest scores from the most relevant DTMS tests were correlated 
with final course grades (A - 4, B « 3, etc.). An attempt was made 
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to use an additional criterion score with a wider range than grades, 
hence possibly yielding higher correlations with the DTMS predictors. 
Thus, instructors were asked to provide course scores reflecting point 
totals for the semester or numerical scores on the final exam. The 
correlations of these course scores with the predictors were nearly 
identical to the correlations for grades, and therefore are not included 
in this report. An attempt also was made to increase the reliability of 
the predictors by summing scores from two DTMS tests when course pre- 
requisites appeared to be relevant to more than one test. Somewhat more 
surprisingly, this attempt was also unsuccessful. Despite the doubling 
in the test length, correlations of grades with one of the DTMS tests was 
usually about as high (or in some cases higher) than the correlation 
with the composite. Composite scores were not created for all insti- 
tutions, hence the occasional blanks for the composite score on the 
following tables. 



Predictive validity coefficients are presented in Tables 5.1 to 5.6. 
Courses reporting grades for fewer than 15 students were not included in 
the tables. Median correlations across courses were in the .40s and 
•50s. Focusing on the 20 college courses with the largest samples (50 or 
more students), in only three was the correlation of the most relevant DTMS 
test with grades less than .40, and in 9 out of the 20 it was greater 
than .50. 



Results 




Table 5.1 

Correlations of Grades with Arithmetic Skills (AS) Scores 



fnr nnufHftH wl t\\ 

l vi L nun w i i.H 


a Mnlor Arlthmotl 


[ r Dnninonflnt" 


Co lie qp 


N 


r with AS 


HH 


48 


.52 


L* 


262 


.68 


LL 


64 


.59 


B 


32 


.33 


DD 


30 


.52 


D 


374 


.57 


G 


25 


.62 


MM 


47 


.66 


NN 


28 


.51 



^Correlation with final exam score; grades not available 
at College L. 
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Table 5.2 

Correlations of Grades with Arithmetic Skills (AS) and 
Elementary Algebra Skills (HAS) Scores Cor 
Elementary Algebra Courses 



College 


N 


r with AS 


N 


r with EAS 


N 


r with AS 


H 


18 


.55 


18 


.37 


18 


.50 


L* 


75 


.54 


69 


.36 


67 


.55 


JJ 


166 


.46 


196 


.54 


162 


.53 


LL 


24 


.55 


34 


.51 


24 


.61 


uu 


HZ 


. HO 


42 


.46 






ii 


37 


.21 


50 


.41 


37 


. 36 


p 


88 


.64 


107 


.70 






G 


24 


.33 


59 


.36 






R 


231 


.27 


203 


.47 


203 


.46 


0 


52 


.14 


53 


.27 


49 


.21 


MM 


28 


.44 


33 


.68 






NN 


24 


.38 


32 


.40 






*Correlation 


with 


final exam 


score; grades not available at 


College L. 
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Table 5.3 

Correlations of Grades with Elementary Algol) ra Skills (EAS) 
Scores for Intermediate Algebra Courses 



College N r with EAS 

HH 46 .62 

H 49 .39 

LL 19 .45 

II 79 .46 

Q 47 .41 

NN 19 .25 



Table 5.4 

Correlations of Grades with Elementary Algebra Skills (EAS) and 
Intermediate Algebra Skills (IAS) Scores 
for College Algebra Courses 



College 


N 


r with EAS 


N 


r with IAS 


N 


r with EAS 


HH 


46 


.53 


42 


.37 


42 


.50 


E 


16 


.44 


23 


.52 


15 


.47 


LL 


53 


.30 


60 


.49 


39 


.45 


B 


99 


.51 


141 


.47 






S 






99 


.33 






P 


35 


.58 


33 


.68 






Q 


25 


.54 


25 


.50 






NN 


19 


.44 


25 


.35 







40 
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Table 5.5 

Correlations of Grades with Intermediate Algebra 
Skills (IAS) Scores for Precalculus Courses 



College N r with IAS 

T 145 .45 

E 47 .55 

B 32 .48 

G 30 .21 

0 312 .49 

KK 26 .58 



Table 5.6 

Correlations of Grades with Intermediate Algebra Skills (IAS) 
and Functions & Graphs (FG) Scores 
for Calculus Courses 



College 


N 


r with IAS 


N 


r with FG 


N 


r with IAS 


T 


114 


.55 


105 


.52 


105 


.59 


E 


22 


.36 


21 


.24 


21 


.37 


B 


54 


.61 


41 


.51 






S 


85 


.38 


83 


• 44 


80 


.46 


II 


24 


.75 


24 


.67 


24 


.77 
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Results were more mixed for courses outside the general algebra- 
calculus sequence. The Elementary Algebra Skills test was a good predictor 
(r - .54; N ■ 104) of grades in a "Technical Math" course at a New 
England vocational-technical college (College EE). And the Intermediate 
Algebra Skills test was a reasonably good predictor of grades in "Finite 
Mathematics" (at College B, r - .28, N = 50; at College F, r - .45, N - 
73). But, the Intermediate Algebra Skills test was a poor predictor 
of statistics grades (at College E, r ■ -.05, N = 10; at College F, r 
» .12, N « 10). Although little confidence can be placed in this 

result due to the very small sample sizes, it should at least serve as 

« 

a warning to colleges to proceed cautiously when using the DTMS tests 
outside the regular algebra-calculus sequence. This point is also 
relevant for courses in "Business Mathematics." At College LL, the 
Elementary Algebra Skills test was unsuccessful at predicting grades 
in a course titled "Business Mathematics" (r ■ .04; N - 38), but did 
predict grades in a course titled "Mathematics for Business and 
Economics I" at the same institution (r ■ .44; N = 47). Examination 



of the catalogue descriptions of the two courses helps to explain the 
discrepancy. The "Business Mathematics" course is described as, "A 
study of simple and compound interest, bank discount, payrolls, taxes, 
insurance, markup and markdown, corporate securities, depreciation, 
and purchase discounts" while the "Mathematics for Business and 
Economics I" course is described as, "A study of equations, inequalities, 
matrices, linear programming, linear quadratic, polynomial, rational, 
exponential, and logarithmic functions. Applications to business and 
economic problems are emphasized." Clearly, algebra skills are more 
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related to the latter course than to the former. Two other institutions 
had business mathematics courses with descriptions more like the 
latter course described above, and correlations with grades in these 
courses also were similiar (at College S, r = .42, N =» 233; at 
College NN, r =» .39, N - 18). 

Test Score - Grade Cross-tabulations 

Although the correlations provide a good general index of the 
relationship between test scores and grades, they provide an incomplete 
picture of the relationship and are not useful in establishing cutoff 
scores. A cross-tabulation of grades and test scores (see Table 5.7) 



Table 5.7 

Cross-tabulation of Course Grades and Elementary Algebra 
Skills Scores for the Precalculus Courses at College 0 

Score Kanrcjt on Grades 
DTMS Elomencary 

Algebra Test ABCDFWFWPWlNo Grade Total 



0-10 
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0 


3 


0 


6 




6 
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0 


2 


18 


11-15 


0 


I 


3 


7 


18 




12 


1 


0 


3 


48 


16-20 


1 


0 


18 


10 


28 




12 


2 


1 


2 


75 


21-25 


5 


20 


22 


20 


30 




12 


3 


1 


0 


115 


26-30 


13 


31 


28 


7 


12 




3 


1 


0 


2 


98 


31-35 


9 


4 


6 


1 


X 


0 


1 


0 


0 


0 


22 


)tal 


28 


56 


80 


45 


95 


8 


46 


7 


2 


9 


376 
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VF - Withdrew failing 
WP - Withdrew passing 
W - Withdrew 
I - Incomplete 
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provides information that is much more useful* If College 0 (a campus of 
a major state university in a southeastern state) wished to use the 
Elementary Algebra Skills test to select students for their precalculus 
course or to advise students to expect difficulties, the data on Table 
5.7 suggests that a score of 20 might be a reasonable cutoff. Only 
2 students with scores of 20 or below received an A or a B in the course 



while 57 students with a score of 20 or below received an F or withdrew 
while failing. On the other hand, 91 students with scores above 20 
received an A or a B, while 46 students with scores in this range 
received an F or withdrew while failing. Of course, cutting scores 
could be put at any number of different score levels depending on the 
social costs of excluding students who have some reasonable chance to 
succeed relative to the costs of instructing large numbers of students 
who will probably fail. 

Comparisons with the SAT -Mathematical 

The Scholastic Aptitude Test (SAT) is an excellent predictor of 
general freshman grade point averages. However, it is by design a 
general aptitude test not tied to the specific content of any course, 
and it must cover the full range of abilities. The DTMS tests, by way of 
contrast, are closely linked to mathematics course content, and any given 
test is designed to assess only a limited number of skills. Thus it was 
predicted that the DTMS tests, despite their relatively small number of 
items, might be better predictors of success in specific beginning 
mathematics courses than the SAT-Mathematical. 
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SAT-Mathematical scores from two institutions were available for 
this analysis. One was a two-year private junior college (College 
II) and the other was a four-year public university with an open admissions 
policy (College R) . The institutions used locally developed tests for 
placement. At College II, the initial course was titled "Preparatory 
Mathematics 11 and was primarily a review of elementary algebra. For 
the 48 students from this course with complete data, the mean on the DTMS 
Elementary Algebra Skills test administered at the beginning of the 
semester was 13.68 (SD ■ 4.73); the mean SAT -Mathematical score was 
344.8 (SD = 53.9). The standard deviation indicates that the restric- 
tion in range is not so severe as to preclude the possibility of a 
substantial correlation between grades and SAT-Mathematical scores. 
However, the actual correlation was very low (r « .04). The correlation 
of the Elementary Algebra Skill test with grades was considerably greater 
(r ■ .41). The difference between these correlations is statistically 
significant (z « 2.16; p < .05) according to the test described by 
McNemar ( 1949, pp. 124-125). The cprrelation between SAT-Mathematical 
scores and Elementary Algebra Skills scores was .25. A similar, though 
not quite as dramatic, difference in correlations was noted for the 
elementary algebra course at College R. For the 198 students with 
complete data, the mean Elementary Algebra Skills score was 15.68 
(SD - 5.03) and the mean SAT -Mathematical score was 385.5 
(SD - 65.3). The correlation of the SAT-Mathematical with grades was 
•26 and the correlation of Elementary Algebra Skills with grades was 
.47. Again, the correlation was significantly greater for the DTMS test 
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(z - 2.94; p < .02). The correlation of SAT-Mathematical and Elementary 
Algebra Skills was .38. 

For the 73 people with complete data in the elementary algebra 
course at College II, the mean SAT-Mathematical score was 383.7 
(SD - 78.1) and the mean score on the Elementary Algebra Skills test 
was 18.49 (SD - 5.59). The correlation of the SAT -Mathematical scores 
with grades was .21 and the correlation of the Elementary Algebra Skills 
test with grades .46; the difference between these correlations with this 
small sample and the conservative two-tailed test is not significant 
(z - i.94; p > .05), although with the possibly justifiable one-tailed 
test it would be significant. The correlation of SAT -Mathematical scores 
and Elementary Algebra Skills scores was .30. 

For the 41 students with complete data in the "Mathematical Analysis 
course at College II, the mean SAT-Mathematical score was 414.4 
(SD ■ 67.4) and the mean Intermediate Algebra Skills score was 12.74 
(SD - 4.88). Correlation with grades for the SAT and DTMS scores were 
.18 and .52, respectively. These correlations are significantly different 
(z « 2.23; p < .05). The correlation of the SAT-Mathematical and Intermed 
iate Algebra Skills scores was .48. 

The predictive value of the SAT -Mathematical is much more apparent in 
the two more advanced courses at College II. Mean SAT -Mathematical 
scores were 454.3 (SD « 79.6; N - 40) in the elementary functions 
course and 562.1 (SD « 66.9; N « 19) in the calculus course. Corre- 
lations of SAT -Mathematical scores with grades in these two courses were 
.39 (elementary functions) and .55 (calculus). Although the SAT* scores 
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have more predictive validity for these more advanced courses, this 
should not imply that the DTMS teats have any less validity. The corre- 
lation of the Intermediate Algebra Skills score with grades was .37 for 
the elementary functions course and .75 for the small sample in the 
calculus course. Mean scores on Intermediate Algebra Skills in the two 
courses were 15.07 (SD * 4.58) and 22.68 (SD = 5.24). The mean score 
on the Functions & Graphs test for students in the calculus course was 
17.13 (SD » 3.79) and its correlation with grades was .67. 

For remedial level college mathematics courses, predictions 
of course grades from the DTMS tests are clearly superior to predictions 
from SAT -Mathematical scores. For more advanced courses, the DTMS tests 
appear to predict grades a;- least as well as scores from the SAT. The 
superiority of a test designed to assess certain specific skills over a 
general aptitude measure when predicting a criterion that is closely 
related to those skills was also noted by Breli:nd (1977). He observed 
that TSWE scores provided better predictions of es&ay test performance 
at the end of an English course than did SAT -Verbal ccores. 

Regression Analyses 

Although a demonstration of predictive correlation provides some 
information on the validity of the DTMS teste, it is of little practical 
significance if it cannot significantly improve predictions made from 
other information already in a students" record. Relatively large sample 
sizes are needed for these multivariate analyses, hence only regressions 
for a fev of the largest courses will be presented here. 
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At College D, grades in high school algebra correlated .38 with 
grades for the 203 students in the 11 Introductory Mathematics" course. 
This course included some arithmetic review and some elementary algebra. 
Could adding scores from the Arithmetic Skills test to the high school 
grades significantly improve the correlation? The answer was a clear 
yes, with the multiple correlation increasing significantly (p < .01) to 
.58. The correlation from the Arithmetic Skills test score alone was .54 
so also including the high school algebra mark in a prediction equation 
is of only marginal utility. Standardized regression weights were .22 
for the high school grade and .46 for the Arithmetic Skills test score. 

For the 295 students in the precalculus course at College 0, high school 
algebra grades correlated .24 with grades in the college course. Adding 
scores from the Elementary Algebra Skills tests dramatically improved the 
multiple correlation to .60. 

At College R, grades for the 162 students in the elementary algebra 
course were predicted using both high school grades and SAT-Mathematical 
scores. The correlation for grades alone was .21, adding the SAT score 
significantly improved the correlation to .36. With two variables 
already in the equation, the Elementary Algebra Skills test score still 
significantly (p < .01) improved the multiple correlation to .51. 
Looking at it the other way, the other two variables contribute only 
minimally to the predictions since scores from the Elementary Algebra 
Skills test score alone correlated .46 with grades in the elementary 
algebra course. 
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Summary 

Scores on the DTMS, either by themselves or in combination with 
other data, appear to be good predictors of success in college mathematics 
courses. For remedial courses, they are apparently better predictors 
than SAT scores* For courses outside the general algebra-calculus 
sequence (e.g., some business mathematics courses) DTMS scores may be 
poor predictors of success. 
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6. Course Satisfaction Analysis 

A questionnaire was developed in order to determine whether students 
believed the course in which they were placed (or in which they chose to 
enroll) was too hard, too easy, or at an appropriate level. Specifically, 
they were asked, "For someone with preparation in mathematics similar 
to your own, how would you describe the difficulty level of the course 
you are now finishing?" Five response choices were provided ranging from 
"much too easy" to "much too difficult." A second question asked, "Would 
you recommend this course to someone who had a mathematics background 
similar to yours?" Three response choices were provided: 1) "No, they 
should take a less advanced course first," 2) "Yes," 3) "No, they should 
skip this course and take a more advanced course." It was hypothesized 
that students with relatively high DTMS pretest scores would consider the 
course to be too easy while students with low scores would perceive it as 
too difficult. 

Results 

Questionnaires were returned from the developmental math course 
(arithmetic and some elementary algebra) at College L and from the 
precalculus course at College T. Response rates were very poor (27 out 
of 108 at College L and 37 out of 127 at College T) . It is not known 
whether the low response rate was due to errors in distributing the forms 
or due to the failure of some students to put their responses in the 
appropriate boxes of the answer sheet (responses were to be placed in a 
special column of the DTMS answer sheet used for the posttest). At 
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College L, there is some evidence to suggest that the better students 
responded to the questionnaire* The mean score on the Elementary Algebra 
Skills test was 12.88 (SD - 5.30) in the total group and 15.15 (SD * 6.59) 
in the group that responded to the questionnaire. However, in College T, 
Intermediate Algebra Skills means were nearly identical in the total 
group and in the questionnaire group (19.98 vs. 19.89). In both insti- 
tutions, however, results of this analysis must be treated as very 
tentative. 

The correlation of the combined score from the Arithmetic Skills and 
Elementary Algebra Skills tests with the five-point perceived course 
difficulty score was -.64; consistent with predictions, students with 
high test scores tended to perceive the course as easier than did students 
with low test scores. Eleven students thought the course was "just about 
the right difficulty level," 13 thought it was too easy (3 "much too 
easy" and 10 "a little too easy") and only 3 thought it was too difficult 
(1 "much too difficult" and 2 "a little too difficult"). The correlation 
with the three-point scale indicating whether they would recommend an 
easier or more difficult course for students with preparation in mathe- 
matics similar to their own was .45, indicating that students with high 
test scores were more likely to recommend taking a more advanced course 
first. Only one student actually recommended taking an easier course 
first, so the basic discrimination was between the 16 students who would 
recommend the course they were currently taking and the 10 students who 
thought a more advanced course would be more appropriate. 
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At College T, scores on the Intermediate Algebra Skills test corre- 
lated -•40 with t-.e parceived course difficulty score, and -»03 with 
the "recommend" score. Three of the 37 students who responded to the 
"difficulty" item failed to respond to the "recommend" item, and of 
the remaining 34 students, 24 of them indicated "Yes" they would recommend 
the course they were taki:.^» This generally positive attitude may have 
been responsible for the lack of correlation in this group. The remaining 
10 students were evenly divided between the two "No" categories. A 
cross-tabulation of Intermediate Algebra Skills scores and the course 
difficulty ratings is presented in Table 6.1. 



Table 6.1 

r,ross- tabulation of Intermediate Algebra Skills Scores 
and Perceived Course Difficulty Ratings 
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Of the 11 students who thought the course was too difficult, only 1 had 
an Intermediate Algebra Skills score greater than 21. In contrast, 
over half (12 out of 23) of the students who thought the course diffi- 
culty level was about right or too easy had scores greater than 21. 
Thus, the major prediction was confirmed; DTMS scores were related to the 
difficulty of mathematics courses as perceived by the students taking them. 




7. Trait-Treatment Interaction 

A demonstration that a test can predict which students are not 
likely to succeed if placed directly into a more advanced course (Course 
B) indicates that the test is useful in selecting students for that 
course, but says nothing about the students who were not selected. But 
if the ultimate goal is for all students eventually to succeed in Course 
B , then for a placement test to be of value, it would be useful to 
demonstrate that some students are more successful if placed first into a 
less advanced course and then into Course B (long sequence) while other 
students do at least as well if placed directly into Course B (short 
sequence). In other words, there should be a trait-treatment interaction 
(see Cronbach & Snow, 1977, for a comprehensive discussion of this 
approach) • 

Ideally, at least from an evaluation point of view, students should 
be randomly assigned to the long sequence or short sequence groups. 
Course B should be exactly the same for long and short sequence groups, 
and there should be at least 100 students in each group. Unfortunately, 
it was not possible to find an institution meeting these criteria. Indeed, 
for the institution selected as offering the best approximation to a good 
trait-treatment interaction study (the precalculus-calculus sequence at 
College T) , none of the above conditions were met. Instead of random 

The reader should keep in mind that this is not always the ultimate 
goal; some students with very low placement test scores may decide to 
enroll in programs that would never require them to take Course B. 
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assignment, students were assigned to the long sequence (precalculus then 
calculus) or the short sequence (calculus only) on the basis of a locally 
developed placement exam, or by having had calculus in high school, or by 
having had a precalculus course in college. Students in the short 
sequence took calculus in the fall, while students in the long sequence 
took it in the spring. One must assume that the course was essentially 
identical in the fall and spring in terms of the quality of instruction 
and the material covered. For courses in some subject areas this assump- 
tion is clearly untenable, but it is probably not too unreasonable for 
large freshman sections of a standard calculus course in which the same 
textbook is used from one semester to the next and the mathematics 
department attempts to maintain a reasonably consistent standard. 
Because math grades are used as the criterion, it is also necessary to 
assume consistent grading standards across semesters. Again, this would 
not be a reasonable assumption in many courses, but it is probably not 
unreasonable for calculus courses in which grading is based on objective 
exams that are very similar from one semester to the next. Sample sizes 
were 114 in the short sequence group and 61 in the long sequence group. 
Although this provides less power than the 100 subjects per treatment 
recommended by Cronbach and Snow (1977, p. 46), it is at least better 
than the 40 or fewer subjects per treatment that Cronbach and Snow assert 
is typical for studies of this type. 

Results 

With grades in calculus as the criterion (Y) and scores on the 
Intermediate Algebra Skills test administered at the beginning of the 



fall semester as the predictor (X), the regression line for students in 
the short sequence was Y - .212X - 2.80. Their mean on X was 25.93 
(SD - 3.35) and their mean on Y was 2.71 (SD = 1.29). The comparable 
line for students in the long sequence was Y » .059X + 1.01. The mean 
on X for long sequence students was 20.87 (SD ■ 4.16) and their mean on 
Y was 2.23 (SD - 1.09). Although the means in the two groups were clearly 
different, there was also some overlap in the distributions. Scores in 
both groups had a wide range (10 to 29 for the long sequence and 15 to 30 
for the short sequence). Slightly over 20% of the students in the long 
sequence group had pretest scores of 25 or more, while 23% of the students 
in the short sequence group had scores of 24 or less. The regression 
lines are presented in Figure 7.1. Rather than a linear fit, the long 
sequence line should probably become steeper at the upper end. Of the nine 
students in this group with scores over 25, two received a grade of C in 
calculus, six received Bs and one received an A. 
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Figure 7.1. Trait-treatment interaction between Intermediate Algebra 
Skills scores and calculus grades. 
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The F-test of the difference in the slopes was significant (F = 11.67, 
df « 1,171, p < •001). Although the F-test lacks precise meaning because 
the two groups were not randomly sampled from the same population, the 
magnitude of the interaction suggests that there is a true differential 
effect for the longer sequence. 

The regression lines intersect at an Intermediate Algebra Skills 
score of about 25, hence for scores around this level it would seem to 
make little difference which sequence students took. However, students 
with scores around 18 on the pretest would apparently be much better off 
in the long sequence where their predicted grade in calculus is a C 
whereas their predicted calculus grade would be a full letter grade lower 
if they skipped the precalculus course. This interpretation is consis- 
tent with the gain analyses presented in Chapter 3 in which it was 
demonstrated that the precalculus course at College T was quite effective 
in increasing students' skills in intermediate algebra. 
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8. Content Validity 

College level mathematics courses typically form a closely linked 
segmented sequence in which success in later courses is assumed to depend 
on mastery of the content of the earlier courses (Willingham, 1974) • 
Placement into the appropriate course in the sequence could then be done 
with a test that was a valid reflection of the content of the preceding 
course. The analysis described below was an attempt to yield some data 
on the content validity of the DTMS. 

A questionnaire was developed in which faculty members were asked 
to rate items on the DTMS "test or two tests that most nearly match the 
content of your course*" For each item, the instructors were asked to 
"make a judgment about the importance of the skill assessed by the item 
as a prerequisite for your course, i.e., how important it is for the 
student to possess that skill before beginning your course." Ratings 
were to be made on a three-point scale (1 - relatively unimportant, 2 * 
moderately important, and 3 ■ very important). They were also asked to 
rate "how important the skill is as an objective of your course, i.e., 
how important it is for the student to have attained the skill by the end 
of your course." The same rating scale described above was used again. 
Instructors were then asked "to make a rough judgment of the amount of 
class time (in minutes) that is spent on the skill assessed by the item." 
Since responses to this time question tended to vary widely from instructor 
to instructor, it was judged to be too unreliable to be used in the 
analyses. 
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Every item on a specific DTMS test might be related to important 
course objectives, but if these objectives represented only a small 
proportion of the total set of objectives for the course, then the test 
might not be very useful for placement purposes* Therefore, each instructor 
was asked "to make a global judgment of the percentage of important 
course objectives assessed by the test as a whole." Other items on the 
questionnaire requested information about course prerequisites and 
placement procedures* A copy of the questionnaire is in Appendix B. 

The questionnaires were mailed to each institution in the study. 
The coordinator in each school was asked to distribute the questionnaires 
to "each instructor of a freshman mathematics course that is open to 
beginning students even though it may also contain substantial numbers 
of more advanced students"; instructors of remedial or developmental 
courses were also to be included. 

Questionnaires were returned from faculty members representing 126 
different courses. The faculty members were asked to classify the 
courses with the term that "best describes your course." Twelve categories 
were supplied (see Item 1 on the questionnaire in Appendix B) with an 
"other (specify) 11 category for courses not listed. The "geometry" and 
"analytic geometry 11 categories were not used at all, and 11 instructors 
used the "other" category. Thus, 115 courses were described with 10 of 
the provided descriptors. Only categories represented by responses from 
at least 12 instructors are presented below. These include arithmetic, 
elementary algebra, intermediate algebra, college algebra, and calculus. 
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Arithmetic Courses 

Fourteen faculty members, 4 from two-year schools and 10 from 
four-year schools, used the term "arithmetic" to describe their courses. 
All of the courses were described as "remedial, compensatory, or develop- 
mental." For nine of the courses, students were required to enroll in 
the prescribed courses, while placement was advised for the other five. 
Half of the respondents indicated that the Arithmetic Skills test alone 
most nearly matched the content of their courses, while the other half 
indicated that the combination of the Arithmetic Skills and the Elementary 
Algebra Skills tests provided the best match. 

Arithmet ic Skills Test: Prerequisite/Objective ratings . Re s p ons es 
to the "importance as a prerequisite" and "importance as an objective" 
items suggested that the Arithmetic Skills test is well suited to the 
arithmetic courses sampled. A,i shown in Table 8. i, none of the 35 items 
was rated as "very important" at a prerequisite by as mai.. as half of the 
respondents. In contrast, 3i items v;^re rated as 'very imports^*; 11 as 
objectives by at least half of the respondents. The "nean item response," 
on the table is the sum of the number oi faculty responses in a category 
(i.e., very important as a prerequisite) divided by the nimber of items 
on the test. It provides an indication of the distribution of faculty 
responses for the test as a whole. 

Arithmetic Skills T^s t: Coverage of course c o ntent . Of the 10 
instructors who responded to the item on the percentage of important 
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Table 8.1 
Arithmetic Courses 

Arithmetic Skills Test 
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course objectives assessed by the Arithmetic Skills test, all indicated 
that more than 50% of their important objectives were assessed. Table 
8.1, Question 1 shows the distribution of percentage of objectives 
assessed by the Arithmetic Skills test. Written responses by some 
faculty members noted that their course objectives not covered by the 
test include metric system problems, operations with negative numbers, 
factoring, and exponents. The questionnaire responses indicated that the 
items on the Arithmetic Skills test are closely linked to the objectives 
of these college arithmetic courses, and provide reasonably complete 
coverage of their most important objectives. 

Elementary Algebra Skills Test: Prerequisite/Objective ratings . 
The seven arithmetic instructors who indicated that the Elementary 
Algebra Skills test also matched the content of their courses were all 
from four-year colleges. As would be expected, very few of the items on 
this test were seen as important prerequisites by the arithmetic instructors. 
Eight of the 35 items were marked as very important as an objective by 
half or more of the respondents. (See Table 8.2.) Five of the eight 
items were from Cluster A — Operations with Real Numbers. Two of the 
items were selected as very important objectives by all seven respondents. 
Both of these items involved manipulations (addition, subtraction, 
multiplication, and division) with negative numbers, and unlike most 
other items in the Elementary Algebra Skills test, neither item involved 
solving an equation for an unknown quantity. 

Elementary Algebra Skills Test; Coverage of course content . Six of 
the seven faculty members responded to the item concerning the percentage 
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Table 8.2 
Arithmetic Courses 

Elementary Algebra Skills Test 
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of important course objectives assessed by the Arithmetic Skills,, and 
Elementary Algebra SkillB tests combined. Table 8.2, Question 2, reports 
the distribution of responses. Objectives noted by respondents as not 
assessed by the test are systems of measurement, geometry, graphs, 
and simple statistics (i.e., mode, median, and mean). 

Taken by itself, the Elementary Algebra Skills test has little 
relevance to arithmetic courses. But, for some arithmetic courses, some 
of the items on this test, especially those in Cluster A, may be useful 
additions to the Arithmetic Skills test. In general, institutions 
wishing to use both tests to yield scores that are adequate reflections 
of the content of their arithmetic courses should review the Elementary 
Algebra Skills test items carefully. Only responses on a relatively 
small number of items judged to be relevant should be included in computing 
total scores. Thus institutions could instruct students to respond only 
to Cluster A on the Elementary Algebra Skills test. However, it may be 
more practical simply to rely on scores from the Arithmetic Skills test 
even though a few areas covered only on the Elementary Algebra Skills 
test would have to remain unassessed. 

Elementary Algebra Courses 

Thirty-six respondents, 19 from two-year schools and 17 from four-year 
schools, chose elementary algebra as the term that best described their 
courses. The majority of those courses, 30 or 83% overall, were classified 
as remedial, compensatory, or developmental by their instructors. Five 
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couraes wore described as regular and one courue was reported to be 
advanced or honors. In response to the item Inquiring about prerequisites 
19 of the respondents reported no prerequisites for their elementary 
algebra courses; 13 reported that one year of high school algebra or its 
equivalent was prerequisite for their courses. The options "two years of 
high school algebra" and "at least ten weeks of trigonometry" were not 
chosen. Three faculty members circled the "other" category and specified 
that basic arithmetic skills were prerequisite. When responding to the 
item inquiring whether or not students were required or advised to enroll 
in their courses, 17 faculty members reported their courses were required 
while another 17 indicated that students were advised to enroll in their 
courses. Two respondents reported that, in some cases, students were 
required to enroll, while in other cases, students were only advised to 
enroll. 

When asked which test(s) best matched the content of their courses, 
none of the respondents chose the Arithmetic Skills test alone; however, 
the majority of respondents, 21 chose the combination of the Arithmetic 
Skills Test and the Elementary Algebra Skills test. Ten faculty responden 
reported that the Elementary Algebra Skills test alone best matched the 
content of their courses. Another five respondents chose the combination 
of the Elementary and Intermediate Algebra Skills tests as representing 
the best course content match. 

Elementary Algebra Skills Test; Prerequisite/Objective ratings . 
The Elementary Algebra Skills test, either by itself or in combination 
with its antecedent or subsequent test, was chosen by 100% of the respond- 
ents as the test that best matched the content of their elementary 
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algebra courses. Of the 35 items on the Elementary Algebra Skills test, 
32 were rated by half or more of the respondents as being unimportant 
prerequisites. In contrast, none of the items was rated by more than 10 
respondents as a very important prerequisite. (See Table 8.3.) Thus, 
the Elementary Algebra Skills test does not measure skills that students 
are already expected to have mastered before entering elementary algebra 
courses. 

All 35 items were rated by at least 52% of the elementary algebra 
instructors as very important objectives. The mean item response for the 
very important as an objective category was 27.9 respondents out of a 
possible total of 36. These responses strongly indicate that the Elementary 
Algebra Skills test does assess skills that students should have mastered 
by the end of elementary algebra courses. 

Elementary Algebra Skills Test: Coverage of course content • 
Thirty-three of the 36 faculty members reported the percentage of important 
course objectives assessed by the Elementary Algebra Skills test. (See 
Table 8.3, Question 1.) Thirty-two faculty members indicated that 50% or 
more of their course objectives were assessed by the test. The one 
faculty member who reported less than 50% commented that the test did not 
adequately address "simple algebraic operations, such as simple equations, 
hierarchy of operations, and simplification. 11 Nevertheless, that faculty 
member still reported that 50% of the course objectives were assessed by 
the combination of the Arithmetic Skills and Elementary Algebra Skills tests. 
Fifteen of the respondents indicated that more than 80% of their course 
objectives were assessed. Course objectives noted by respondents as not 
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Tobla 8. 3 
Elementury Algebra Couraea 

Elementary Algebra Skills Teat 
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assessed by rtu* Momentary Algubra Skills tear Included elementary 
geometry, quadratic equations, graphing • linear equations, algebraic 
fractions, fractional exponents, radical equations, and absolute values. 
Two respondents who chose the Arithmetic Skills and Elementary Algebra 
Skills tests commented that items on the Intermediate Algebra Skills test 
assessed some course objectives not assessed by the other two tests, but 
that it is too time consuming to administer all three tests. One of them 
suggested combining the content of the three tests into two tests. 

Arithmetic Skills Test: Prerequisite/Objective ratings . The 21 
faculty members who chose the combination of the Arithmetic Skills and 
Elementary Algebra Skills tests as the combination that best represented 
the content of their courses, all overwhelmingly reported that the 
majority of items on the Arithmetic Skills test represent very important 
prerequisites of their elementary algebra courses. The mean item response 
for the very important as a prerequisite category was 14*1 out of the 
total of 21 respondents. Complementary to this statistic is that the 
mean item response for the very important as an objective category was 
8.6 out of 21. These two statistics indicate that the content of the 
Arithmetic Skills test is seen as covering very important prerequisite 
material of the majority of these courses. (See Table 8.4») 

Arithmetic Skills Test: Coverage of course content . Sixteen of the 
21 respondents answered the question asking about the percentage of 
course objectives assessed by the Arithmetic Skills test. Eleven faculty 
members reported that it assessed 50% or less; five respondents reported 
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Table 8.4 
Elementary Algebra Courses 

Arithmetic Skills Test 

TEST H 1 — CUMBINCD SCHCCLS (N= 21) 
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that more than 80% of their course objectives were assessed. (See Table 
8.4, Question 1.) Table 8.3, Question 2, shows that 19 of the 21 
faculty members who selected the combination of the Arithmetic Skills and 
Elementary Algebra Skills tests indicated that more than 50% of their 
course objectives were assessed by these combined tests. Fourteen of 
them indicated that more than 80% of their objectives were assessed by 
these two tests. In combination then, the Arithmetic Skills test, 
covering mainly prerequisite material, and the Elementary Algebra Skills 
test, covering mainly course objectives, appear to be appropriate tests 
for the elementary algebra courses sampled. 

Intermediate Algebra Courses 

Intermediate algebra was selected by 12 respondents, 6 from two-year 
schools and 6 from four-year schools, as the term that best described 
their courses. Five of those courses were rated as remedial, compensatory, 
or developmental and seven were reported to be regular. Ten faculty 
respondents indicated that one year of high school algebra or its equivalent 
was prerequisite to their courses. Two respondents from the same institutic 
marked the other category ant ■?» dfied the following: (a) admission as 
a regular, not special studies, student in the institution, and (b) 
attainment of a cutoff score on the SAT or on a local test. 

In responding to the item asking them to choose the DTMS test(s) 
that most nearly matched the content of their courses, one instructor of a 
remedial course circled the Elementary Algebra Skills test alone. Seven 
faculty members, two of remedial courses and five of regular classes, 
circled the combination of the Elementary Algebra and the Intermediate 
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Algebra Skills tests. Three respondents, one of a regular course and two 
of remedial classes, chose only the Intermediate Algebra Skills test. 
One instructor of a regular course selected the combination of the 
Intermediate Algebra Skills test and the Functions & Graphs test. 

Elementary Algebra Skills Test: Prerequisite/Objective ratings . 
Responses to the importance as a prerequisite and importance as an 
objective item indicate that the content of the Elementary Algebra Skills 
test is well matched to the objectives of the intermediate algebra 
courses represented by the eight respondents who chose it. Of the 35 
items, 31 were rated by five or more respondents as representing content 
that is very important as objectives in their courses. The mean item 
response for the very important as an objective category was 5.7 out 
of a possible 8 respondents. (See Table 8.5.) 

In contrast to this is that none of the 35 items were rated by more 
than four respondents as being very important prerequisites. Seventeen 
questions were not rated by any respondents as representing very important 
prerequisites • 

Elementary Algebra Skills Test: Coverage of course content . Of the 
six faculty members who responded to the question inquiring about the 
percentage of course objectives assessed by the Elementary Algebra Skills 
test alone, three, two instructors of remedial courses and one instructor 
of a regular class, reported that it assessed 50Z or less, although not 
less than 30%. The other three, all instructors of regular classes, 
reported that it assessed 80% or more of their course objectives. (See 
Table 8.5, Question 1). All six respondents were ones who had chosen 

71 
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Table 8.5 
Intermediate Algebra Courses 

Elementary Algebra Skills Test 

TEST H 2 -- CUMBINfrC SCH(.CLS (N= 8) 
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the Elementary Algebra Skills test In combination with the Intermediate 
Algebra Skills test as the tests that best matched the content of their 
courses. Two respondents who chose that combination did not report the 
percentage of objectives assessed, but one of them noted that logic and 
sets were not covered by the Elementary Algebra Skills test or the 
Intermediate Algebra Skills test. 

These data indicate that the content of these eight courses, described 
as intermediate algebra courses by their instructors, is assessed to some 
extent by the Elementary Algebra Skills test; however, the Elementary 
Algebra Skills test alone does not always assess the majority of objectives. 

Intermediate Algebra Skills Test: Prerequisite/Objective ratings . 
Of the 11 respondents who reported that the Intermediate Algebra Skills 
test matched the content of their courses, seven did so in combina- 
tion with the Elementary Algebra Skills test; three reported so alone, 
and one reported it in combination with the Functions & Graphs test. 
The 11 respondents together reported that the items on the Intermediate 
Algebra Skills test were unimportant as prerequisites, but that they were 
very important as objectives of their courses. The responses to the 
objectives rating task revealed that 16 of the 30 items were rated by half 
or more of the 11 respondents as assessing very important course objectives. 
When taken together, the moderately important and very important as 
objectives categories were chosen by at least half of the respondents for 
all but one item* (See Table 8.6.) These responses indicate that the 
items on the Intermediate Algebra Skills test match some of the content 
objectives of the 11 courses described as intermediate algebra by their 
instructors* 
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Table 8.6 
Intermediate Al^sbra Course' 

Intermediate Algisbn Sl-illa Teat 

TEST #3 — COMBINED SOHCCLS (N= U) 
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Intermw^Iate Algebra Skills Test; Coverage of course content * Of 
the seven respondents who chose the Intermediate Algebra Skills test and 
the Elementary Algebra Skills test as the combination that best matched 
the content of their courses, two reported that the combination assessed 
more than 90% of their course objectives; two reported they assessed 81% - 
90% of course objectives; another two indicated they covered 71% - 80% 
of course objectives; and one judged that the two tests together assessed 
70% of the course objectives. (See Table 8.6, Question 2). 

Two of the three respondents who chose only the Intermediate Algebra 
Skills test as the best match reported that 61% - 70% of their course 
objectives were assessed by the test. The third respondent in this group 
did not reply to the item. It appears that the Intermediate Algebra 
Skills test assesses better than 50% of the course objectives of these 
courses. 

The one respondent who chose the Intermediate Algebra Skills test 
in combination with the Functions & Graphs test reported that the 
intermediate level test assessed 51% * 60% of his course objectives, but 
that these two tests combined assessed 81% - 90% of the course objectives* 

All 11 respondents who chose the Intermediate Algebra Skills test 
reported that it alone, or in combination with the preceding or subsequent 
test , assessed at least 50% of their cour ?e objectives • Some of the 
objectives not assessed by the Intermediate Algebra Skills test were 
logic, sets, quadratic equations, evaluation of functions and composite 
functions, absolute value inequalities, formula evaluation, and manipulation 
of vectors. One respondent noted that more emphasis on word problems was 
needed. 
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College Algebra Courses 

College algebra was the term chosen by 15 respondents to describe 
their mathematics courses. Thirteen of these faculty members were 
from four-year schools and two taught at two-year institutions. Three 
respondents from four-year schools described their courses as remedial, 
compensatory, or developmental and 12 respondents, two from two-year 
schools and ten from four-year schools, described their courses as 
regular. Six faculty members (40%) reported that one year of high school 
algebra or its equivalent was prerequisite for their courses. Nine 
respondents (60%) indicated that two years of high school algebra or its 
equivalent were necessary prerequisites for their courses. Twelve 
faculty members (80%) replied that students were advised to enroll in 
their courses and three faculty members reported that their courses were 
required of their students. Ten respondents chose the combination of the 
Elementary Algebra Skills test and the Intermediate Algebra Skills test 
as the best match for their courses. The Intermediate Algebra Skills 
test and the Functions & Graphs test were chosen by five respondents as 
the best match. In both categories, one respondent was from a two-year 
school and the rest were from four-year institutions. 

Elementary Algebra Skills Test: Prerequisite/Objective ratings . None of 
the 35 items was rated by more than six respondents as a very important 
prerequisite, and only five questions were rated by as many as six respondents. 
However, all but two items were rated by at least 80% of the respondents as 
either very important or moderately important as prerequisites. All 35 items 
were rated by 60% or more of the 10 respondents as representing content 
very important as an objective of their courses. (See Table 8.7.) 
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Table 8.7 
College Algebra Courses 

Elementary Algebra Skills Test 



TEST # 2 
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Elementary Algebra Skills Test: Coverage of course content . Eight 
of the 10 respondents who chose the Elementary Algebra Skills test 
reported the percentage of course objectives assessed by this test. Six 
reported it assessed 50% or less and two marked 51% or better. (See 
Table 8.7, Question 1.) 

For the ten courses described, the Elementary Algebra Skills test 
assesses some course objectives, but it also includes content regarded as 
moderately important to very important as prerequisites by a majority of 
respondents. In 60% of these cases, this test alone assessed 50% or less 
of the course objectives. It should be noted, however, that this test 
was not chosen alone, but only in combination with the Intermediate 
Algebra Skills test. 

Intermediate Algebra Skills Test: Prerequisite/Objective ratings . 
The Intermediate Algebra Skills test was chosen by 15 faculty members. 
Of these 15, ten chose it in combination with the Elementary Algebra 
Skills test and five combined it with the Functions & Graphs test. 
Twenty-eight of the 30 items on the Intermediate Algebra Skills test were 
rated by eight or more respondents (more than 50%) as representing very 
important course objectives. The mean item response for this category 
was 10 out of a possible 15 respondents. In contrast, only one item was 
rated by eight or more faculty as a very important prerequisite and it 
was so rated by nine instructors. The mean item response for the very 
important as a prerequisite category was only 3.5 or slightly more than 
20% of the respondents. (See Table 8.8.) 

Intermediate Algebra Skills Test: Coverage of course content . 
Seven of the 10 faculty who chose the Intermediate Algebra Skills 
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Table 8.8 
College Algebra Courses 

Intermediate Algebra Skills Test 



TEST U 3 — COMBINED SCHOOLS (N= 15) 
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test with the Elementary Algebra Skills test reported that the combination 
assessed 61% - 100% of their course objectives. (See Table 8.8, Question 2.) 
One faculty member who chose botn tests reported that 402 of the course 
objectives were assessed. Two faculty who marked both tests did not 
reply to this question. 

Function s & Graphs Test; Prerequisite/Objective ratings . Funct ions 
& Graphs was chosen by five respondents in combination with the Intermediate 
Algebra Skills test. The response pattern to the prerequisite-objective 
racing task shows that all 30 items were strongly rated as unimportant 
prerequisites. The mean item response for this category was 4.6 out of a 
possible 5 respondents. Sixteen questions were rated as very important 
objectives by three or more respondents. (See Table 8.9.) Of interest 
is the fact that all 13 items rated by only one respondent as a very 
important objective were from Cluster C — Trigonometric Functions. 
In fact, these 13 items comprise all of Cluster C and, in all cases, 
these items were ranked by the other four respondents as unimportant 
objectives. No other items were rated as unimportant. (See Table 8.9.) 
These data indicate that, except for the trigonometric items, the items 
of the Functions & Graphs test assess very important objectives of these 
five courses. 

Functions & Graphs Test: Coverage of course content . Four of the 
five respondents reported that the Functions & Graphs test in combination 
with the Intermediate Algebra Skills t est assessed 71% — 100% of course 
objectives. (See Table 8.9, Question 2.) One respondent indicated that 
even together the two tests assessed only 50% of course objectives. This 
faculty member indicated that the important course objectives not assessed 
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Table 8.9 
College Algebra Courses 

Functions & Graphs Test 
TEST * A -- COMBINED SCHCCLS (N= 5) 
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by the two tests were: graphing parabolas (vertex, axis of symmetry, 
and maximum-minimum), absolute value inequalities, solving systems of 



equations using matrices (Gaussian elimination) and determinants (Cramer's 
rule), remainder and factor theorems, synthetic division, and rational roots 
theorem. In general, the Intermediate Algebra Skills and the Functions & 
Graphs tests assess more than 70% of the objectives of 80% of these 
courses. It also should be noted that each of the Cluster C items 
covering trigonometric functions were judged by only one respondent as 
representing important objectives. 

Calculus Courses 

Calculus was chosen by 14 faculty members as the term that best described 
their courses. Thirteen courses were described as regular while one was 
reported to be an advanced or honors course. In regard to prerequisites, 
three respondents reported two years of high school algebra or its 
equivalent and seven reported at least ten weeks of trigonometry as prere- 
quisite. Four chose the other category and specified as follows: 
two faculty reported 2 years of high school algebra plus other courses 
such as 1 year of geometry, 1/2 year of trigonometry, and a course on 
functions and graphs, and analytic geometry; one respondent reported 
that elementary analysis is prerequisite to the course; one reported 
that a strong background in high school mathematics was required. 

Eight faculty members reported that their students were advised to 
enroll in their courses and six indicated that their courses were required. 
Since there is no calculus test in the DTMS, calculus faculty were 
instructed to respond only to the Functions & Graphs test. Only the 
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items on importance as a prerequisite were appropriate, hence they are 
the only items evaluated here. 

Functions & Graphs Test: Prerequisite ratings . Generally, the 
items of the Functions & Graphs test were not overwhelmingly rated as 
prerequisites of the 14 calculus courses* (See Table 8* 10,) The mean 
item response for the three levels of prerequisite were: very important 
as a prerequisite— 5»5; moderately important as a prerequisite — 5*1; and 
unimportant as a prerequisite — 3.3. Although no item was rated as 
unimportant by more than half of the respondents, only five items were 
rated as very important by more than half of the respondents. All five 
of these important prerequisites were in Cluster A (8 items on algebraic 
functions). Thus, the items in Cluster B (9 items on exponential and 
logarithmic functions) and Cluster C (13 items on trigonometric functions) 
were not seen as particularly important prerequisites. 

Summary 

In general, faculty members rated the items on the DTMS tests as 
fairly representative of the key concepts in their courses. However, in 
certain individual cases there were significant discrepancies between 
course content and test coverage. Content validity for any particular 
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Calculus Courses 
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TPrSf #4 — 4-YEAR SCHCCLS (N= 14) 
£B£S£flUI5iI£5 UBJf£Il^ES_ 









-X' 




iU- 




1G1 


1 


7 


6 




5 


2 


102 


4 


4 


6 


8 


•> 


1 


103 


1 


4 


9 


8 


3 


1 


104 


4 


3 


7 


3 


2 


1 


105 


0 


2 


12 


6 


1 


5 


106 


0 


3 


11 


t> 


2 


4 


107 


5 


5 




8 


2 


1 


108 


2 


5. 


7 


3 


6 


3 


109 


5 


b 


4 


H 


2 


1 


no 


4 


b 


5 


9 


1 


1 


111 


0 


4 


10 


7 


2 


3 


112 


6 


3 


5 


9 


0 


2 


113 


2 


7 


5- 


<f 


6 


2 


114 


5 


b 


4 


9 


G 


2 


115 


2 


.10 


2 


4 


5 


3 


116 


b 


7 


2 


8 


1 


2 


117 


4 


9 


1 


7 


3 


2 


118 


5 


4 


5 


8 


1 


2 


119 


4 


8 


2 


6 


3 


3 


120 


4 


3 


7 




1 


2 


121 


4 


7 


3 


1 


4 


3 


122 


2 


't 


8 


5 


1 


6 


123 


4 


9 


1 


4 


b 


3 


124 


5 


3 


6 


9 


1 


1 


12b 


3 


4 


7 


J 


2 


3 


126 


2 


3 


7 


4 


4 


4 


127 


5 


3 


6 


7 


2 


1 


120 


3 




7 


>2 


6 


4 


1 ?9 


A 


ft 


4 


7 


7 


1 



-81- 



9» Summary 

The Descriptive Tests of Mathematics Skills (DTMS) is a group of 
four tests (Arithmetic Skills, Elementary Algebra Skills, Intermediate 
Algebr* Skills, and Functions & Graphs) that was designed to help colleges 
place each admitted student in the appropriate mathematics course. The 
purpose of the study was to determine whether the DTMS tests possess 
characteristics that are generally desirable for tests used to make such 
placement decisions. 

For this project, the DTMS tests were administered in freshman level 
mathematics courses at 36 institutions that represented a broad spectrum 
of two-year and four-year colleger. At some colleges, the tests were 
administered only at the beginning of the semester; in other colleges, 
they were administered both at the beginning and the end of the semester. 

Results from a number of different !;inds of analyses indicated that 
the DTMS tests are potentially very useful in helping to make placement 
decisions* 

(1) Gain Analysis — If a teat is accurately targeted to the 

< utent of a course (and assuming instruction in that course 
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(2) Concurrent Validity — In a sequence of courses where Course X 
is a prerequisite to Course Y, a placement test should be able 
to show when a student has mastered the content of Course X 
and is therefore ready to go directly to Course Y. Demon- 
stration of mastery of course content* as indicated by grades 
in the course, was highly related to scores on the DTMS tests 
administered at the end of the course. 

(3) Predictive Validity — It is often useful to know how well a 
placement test given at the beginning of the semester predicts 
success in a course, as indicated by end of semester grades. 
The DTMS tests were generally very good at making these 
predictions. For most courses* correlation coefficients were 
in tha .40s and .50s. Better predictions were possible from 
DTMS scores than from high school algebra grades, and for 
remedial level courses, better predictions were possible 

from the DTMS tests than from the mathematical section of 
the Scholastic Aptitude Test. 

(4) Relationship to Student Perception of Course Difficulty — 
Students who thought their math courses were easy generally 
had high DTMS scores, and students who thought the courses 
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on the placement test might do better in the advanced course 
if they first take a less advanced course. When the above 
conditions are met, there is a trait-treatment interaction. 
For the current report, a trait-treatment interaction study 
was attempted at only one institution. In the precalculus- 
calculus sequence at that college, there was a significant 
interaction suggesting that the DIMS tests could be a valuable 
tool for placing students in that course sequence. 
(6) Content Analysis — College faculty members in the participating 
institutions rated the DTMS items as providing generally 
good coverage of the key concepts of their courses. 
The data presented in this report provide promising preliminary 
«vi1cnce for the potential value of the DTMS tests for making college 
placement decisions. However, more research is clearly needed as colleges 
begin to make actual placement decisions with DTMS scores. It is only 
through the accumulation of evidence basd on these experiencs that the 
true value and limitations of the DTMS can be determined. 
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APPENDIX A 

Descriptions of Participating Institutions 
Grouped by Region of the Country 



toft Ion: New England 



Col le*e 
Code 



AA 



BB 

CC 

DD 
EE 

FF 



Description of Institution 

two-yenr public tvc' nlcal 
college for men and women 

two-vear public technical 
collegr for men and women 

two-year public technical 
college for men and women 

two-year public community 
college for men and women 

two-year public college 
for men and women 

two-year public technical 
college for men and women 

four-year public college 
for men and women 



L ocation 

small city 

small city 

rural community 

small city 
small city 

suburban community 
suburban community 



Selection 
Ratio 



.80 



Admissions 
Information 



.39 



. 77 



, 36 



,37 



SAT 



Region; Mlddls States 



College 
Code 



Description of institution 

four-year public college 
for men and wcaen 

four-year pur.tr liberal 
arts college . r>r ^<en and 
women 



Location 



ill tc 



suburban community 



Selection 
Ratio 



.68 



.56 



Admission* 
Information 

SAT (V: 41O-500; Q: 440-550) - 
interview recommended 

open admissions for state residents 
SAT for placement 



four-year pub" ' „• college 
for mer and v *n 

four-year public liberal 
arts college for men and 
women 

four-year public liberal 
arts college for men and 
women 

four-year private liberal 
arts college for men and 
women (church affiliated) 



large city 
small town 

1 <rge town 

large town 



.65 



.89 



.86 



SAT (V: 350-500; Q: 350-550) 



SAT 



SAT (preferred; V: 425; Q: 440) or 
ACT 
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Region; Midwestern 



College 
Code 



GG 



HH 
L 



Description of Institution 

two-year public community 
college for men and women 

four-year public university 
foi men and women 



four-year private university 
for men and women (church 
affiliated) 

four-year public college of 
arts and sciences for men 
and women 

four-year public university 
for men and women 

two-year public college 
for men and women 

four-year public university 
for men and women 

four-year public university 
for men and women 

four-year private university 
for men and women (church 
affiliated) 



Location 
suburban community 
rural community 

metropolitan area 
metropolitan area 
small city 

small city 
suburban ommunity 
metropolitan area 



Selection 
Ratio 



1.00 



.70 



.75 



.68 



.64 



.82 



.86 



.85 



Admissions 
Information 

open admissions - no minimum GPA - 
ACT used for counseling/placement 

ACT composite 21 or better for state 
residents - SAT for out-of-state residents 
(also top 502 of high school class for state 
residents) 



SAT or ACT required 



ACT composite 23 or better 
of high school class 



top 50Z 



I 

o 
I 



ACT composite 17 or better for top 50Z - 
ACT composite 22 or better for lower 502 



ACT 



ACT 



ACT or SAT 
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Region; Southern 



College 
Code 



II 



Dn.'irr l p »n of Institution 

two-year private Junior 
college for men and women 
(church affiliated) 



Location 



small town 



Selection 
Ratio 



.<J3 



Admissions 
Information 



SAT (preferred; V: 400-500; Q: 400-550) or 
ACT 



JJ 

0 



KK 



two-year public community 
college for men and women 

four-year public university 
for men and women 

four-year public liberal 
arts and teachers college 
for men and women 

four-year public liberal 
arts college for men and 
women 

two-year public community 
college for men and women 

four-year public university 
for men and women 



suburban community 
small city 
suburban community 

small town 

rural community 
rural community 



1.00 



.70 



.90 



1.00 



open admissions - SAT (preferred) or ACT 
for placement 



SAT (V: 370-460; Q: 400-510) 



.78 



SAT 



open admissions - SAT for placement 



open admissions 



SAT (V: 350-500; Q: 350-500) 



I 
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Ration: SouthwebLern 



College 
Code 



!.L 



MM 



Description of institution 

two-year public community 
(Mlli D e for men and women 

two—e.ir public community 
college for men and women 

two-year public rnmmunity 
college for men and women 



L ocatio n 
suburban comnunity 
suburban community 
suburban communiLy 



Selection 
R.itio 



Admissions 
Information 



open admissions 



open admissions 



open admissions 



Region: Western 



College 
Code 



on 



Description of Institution 

four-vear public university 
for men and women 

two-year public community 
college for men and women 

four-year private university 
for men and women 

four-year public liberal 
arts, technical, business, 
and teachers college for 
men and wof.^n 



location 
small city 
suburban community 
metropolitan arc. 
metropolitan area 



Selection 
Ratio 



.97 



.69 
.99 



Admissions 
Information 



I 

vO 
I 



open admissions - SAT or ACT (preferred; 
ACT composite 15-23) for placement /counseling 

open admissions - TOEFL and Michigan 
Proficiency Test for foreign students 



SAT - ACT 



ACT 



06 
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APPENDIX B 
Content Analysis Questionnaire 

This questionnaire was completed by faculty respondents. A discussion 
of their responses is in Section 8 — Content Validity — of this report. 
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CONTENT ANALYSIS 
OF THE DESCRIPTIVE TESTS OF MATHEMATICS SKILLS 

Thank you in advance for your assistance in completing this analysis of 
the content of the Descriptive Tests of Mathematics Skills (DTrtS) . This is one 
of the key elements in the validation of the DTMS that is being conducted by 
Educational Testing Service for the College Board. The information you provide 
will be ext reiuely valuable in helping us to determine the relevance of DTMS 
content c or particular courses. 

Please do not be misled by the apparent length of this form. It contains 
space to evaluate all four of the DTMS tests, but you will be asked to fill out 
only those parts that pertain to the one or two tests that are most relevant 
to your course. 

You should fill out one of these forms for each mathematics course that 
you teach that is open to beginning freshmen. As soon as you have completed 
the form, please mail it back in the return envelope provided. 

If you have any questions please feel free to call collect. 

Brent Bridgeman 
Research Psychologist 
Educational Testing Service 
(609) 921-9000 EXT 2542 



Name of Institution 

Your Name (Optional) 

Course No. Course Title 



£8 



For the following questions, please indicate your choice by drawing a 
circle around the appropriate number. 



1. Which term best describes your course? 

01 Arithmetic 

02 Business Math 

03 Math for Elementary School Teachers 

04 Elementary Algebra 

05 Geometry (excluding Analytic Geometry) 

06 Intermediate Algebra 

07 Fundamental Concepts (Sets, Logic, Real number systems, Probability, 
and Statistics) 

08 College Algebra 

09 Trigonometry (as a separate course only, not as part of an algebra 
course) 

10 Elementary Functions 

11 Analytic Geometry (as a separate course) 

12 Calculus 

13 Other (Specify) 

2. Which term best describes the level of your course? 

01 Remedial, compensatory, or developmental 

02 Advanced or honors 

03 Regular 

3. Which statement best describes the prerequisites for your course? 

01 None 

02 One year of high school algebra (or equivalent) 

03 Two years of high school algebra ( or equivalent) 

04 At least 10 weeks of trigonometry 

05 Other (Specify) 



4, 



Briefly describe how students are placed into your course. If scores from 
nationwide testing programs (e.g., SAT, ACT, CGP) are used, please indicate 
the cut-off score employed. 
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5. Are students required to enroll in prescribed courses on the basis of the 
placement decision, or are students simply advised to enroll with the ultimate 
decision left to the student. 

1 Required 2 Advised 

6. Quickly scan the four tests in the accompanying booklet to identify the 
test or two tests that most nearly match the content of your course. For 
example, if you are teaching a remedial arithmetic course you might select 
only Test 1 (Arithmetic Skills), but for a college algebra course, you might 
select Tests 3 (Intermediate Algebra Skills) and 4 (Functions and Graphs). 
Since there is no calculus test in the DTMS, calculus instructors should 
select Test 4 (Functions and Graphs), 

Circle the number for the description of the test or tests you have selected 
as most nearly matching the content of your course. 

1 Test 1 (Arithmetic Skills) 

2 Tests 1 and 2 (Arithmetic Skills and Elementary Algebra Skills) 

3 Test 2 (Elementary Algebra Skills) 

4 Tests 2 and 3 (Elementary Algebra Skills and Intermediate Algebra Skills) 

5 Test 3 (Intermediate Algebra Skills) 

6 Tests 3 and 4 (Intermediate Algebra Skills and Functions and Graphs) 

7 Test 4 (Functions and Graphs) 

In the next section you are asked to make three judgements about each item 
in the test or tests you selected above. First, you are asked to make a judgement 
about the importance of the skill assessed by the item as a prerequisite for 
your course, i.e., how important it is for the student to possess that skill 
before beginning your course. Next, you are asked how important the skill is 
as an objective of your course, i.e., how important it is for the student 
to have attained the skill by the end of your course. Finally, you are asked 
to make a rough judgement of the amount of class time (in minutes) that is spent 
on the skill assessed by the item. At the end of each test you are asked to make 
a global judgement of the percentage of important course objectives assessed 
by the test as a whole. Please complete the pages for the tests you indicated 
in item 6, and leave the other pages blank. 
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TEST 1 ARITHMETIC SKILLS 
Circle the number of the response you choose. 



Importance as a 
PREREQUISITE 



Importance as an 
OBJECTIVE 



Number of minutes of 
class time spent on 
this skill 
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Test 1 Continued 
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PREREQUISITE 
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Number of minutes of 
class time spent on 
this skill 
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On the whole, what percentage of the Important objectives of your course are 

assessed by this Arithmetic Skills Test? % Please specify on the bottom of 

this page any important course objectives that were not assessed by this test; if 
you also have selected the Elementary Algebra Skills test as relevant to your 
course, leave the bottom of this page blank. 
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TEST 2 ELEMENTARY ALGEBRA SKILLS 



Circle the number of the response you choose. 



Importance as a 
PREREQUISITE 



Importance as an 
OBJECTIVE 



Number of minutes 
of class time spent 
on this skill 
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Importance as a 
PREREQUISITE 



Importance as an 
OBJECTIVE 



Number of minutes 
of class time spent 
on this skill 
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On the whole, what percentage of tho important objectives for your course 
are assessed by this Elementary Algebra Skills Test:? % 

If you also selected Test 1 (Arithmetic Skills) as relevant to your course, 
what percentage of the objectives for your course are assessed by Tests 1 and 2 
combined? % 

Please specify on the bottom of this page any important course objectives that 
were not assessed by this test (or, if appropriate, by this test and Test 1); if 
you also have selected the Intermediate Algebra Skills Test as relevant to your 
course, leave the bottom of this page blank. 
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TEST 3 INTERMEDIATE ALGKRRA SKILLS 



Importance as a 
PREREQUISITE 



Importance as an 
OBJECTIVE 



Number of minute* 
of class time spent 
on tbis skill 
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On the whole, what percentage of the Important objectives of your course 
are assessed by this Intermediate Algebra Skills Test? % 

If you also selected Test 2 (Elementary Algebra Skills) as relevant 
to your course, what percentage of the objectives for your course are assessed 
by Tests 2 and 3 combined? X 

Please specify oh the bottom of this page any Important course objectives 
that were not assessed by this test (or, If appropriate, by this test and Test 2); 
If you also have selected the Functions and Graphs test as relevant to your 
course, leave the bottom of this page blank. 
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Importance as a 
PREREQUISITE 



Importance as an 
OBJECTIVE 



Number of minutes 
of class time spent 
on this skill 
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On the whole, what percentage of the Important objectives of your course 
are assessed by this Functions and Graphs Test? % 

If you also selected Test 3 (intermediate Algebra) as relevant to your course, 
»hat percentage of the objectives for your course are assessed by Tests 3 and 4 
combined? % 

Please specify on the bottom of this page any important course objectives that 
were not assessed by this test (or, if appropriate, by this test and Test 3). 
If you are teaching a calculus course, leave the bottom of the page blank. 
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